{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Serving Alibi-Detect models\n",
    "\n",
    "Out of the box, `mlserver` supports the deployment and serving of [alibi-detect](https://docs.seldon.io/projects/alibi-detect/en/latest/index.html) models. Alibi Detect is an open source Python library focused on outlier, adversarial and drift detection. In this example, we will cover how we can create a detector configuration to then serve it using `mlserver`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Fetch reference data\n",
    "\n",
    "The first step will be to fetch a reference data and other relevant metadata for an `alibi-detect` model. \n",
    "\n",
    "For that, we will use the [alibi](https://github.com/SeldonIO/alibi) library to get the adult dataset with [demographic features from a 1996 US census](https://archive.ics.uci.edu/ml/datasets/census+income).\n",
    "\n",
    "**Note**: Install `alibi` dataset dependencies and `alibi-detect` for detector configuration"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install alibi alibi_detect"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import alibi\n",
    "import matplotlib.pyplot as plt\n",
    "import numpy as np"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "adult = alibi.datasets.fetch_adult()\n",
    "X, y = adult.data, adult.target\n",
    "feature_names = adult.feature_names\n",
    "category_map = adult.category_map"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "n_ref = 10000\n",
    "n_test = 10000\n",
    "\n",
    "X_ref, X_t0, X_t1 = X[:n_ref], X[n_ref:n_ref + n_test], X[n_ref + n_test:n_ref + 2 * n_test]\n",
    "categories_per_feature = {f: None for f in list(category_map.keys())}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Drift Detector Configuration"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This example is based on the Categorical and mixed type data drift detection on income prediction tabular data from the [alibi-detect](https://docs.seldon.io/projects/alibi-detect/en/latest/examples/cd_chi2ks_adult.html) documentation."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Creating detector and saving configuration"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from alibi_detect.cd import TabularDrift\n",
    "cd_tabular = TabularDrift(X_ref, p_val=.05, categories_per_feature=categories_per_feature)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from alibi_detect.utils.saving import save_detector\n",
    "filepath = \"alibi-detector-artifacts\"\n",
    "save_detector(cd_tabular, filepath)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Detecting data drift directly"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "preds = cd_tabular.predict(X_t0,drift_type=\"feature\")\n",
    "\n",
    "labels = ['No!', 'Yes!']\n",
    "print(f\"Threshold {preds['data']['threshold']}\")\n",
    "for f in range(cd_tabular.n_features):\n",
    "    fname = feature_names[f]\n",
    "    is_drift = (preds['data']['p_val'][f] < preds['data']['threshold']).astype(int)\n",
    "    stat_val, p_val = preds['data']['distance'][f], preds['data']['p_val'][f]\n",
    "    print(f'{fname} -- Drift? {labels[is_drift]} -- Chi2 {stat_val:.3f} -- p-value {p_val:.3f}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Serving\n",
    "\n",
    "Now that we have the reference data and other configuration parameters, the next step will be to serve it using `mlserver`. \n",
    "For that, we will need to create 2 configuration files: \n",
    "\n",
    "- `settings.json`: holds the configuration of our server (e.g. ports, log level, etc.).\n",
    "- `model-settings.json`: holds the configuration of our model (e.g. input type, runtime to use, etc.)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### `settings.json`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%writefile settings.json\n",
    "{\n",
    "    \"debug\": \"true\"\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### `model-settings.json`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%writefile model-settings.json\n",
    "{\n",
    "  \"name\": \"income-tabular-drift\",\n",
    "  \"implementation\": \"mlserver_alibi_detect.AlibiDetectRuntime\",\n",
    "  \"parameters\": {\n",
    "    \"uri\": \"./alibi-detector-artifacts\",\n",
    "    \"version\": \"v0.1.0\",\n",
    "    \"extra\": {\n",
    "      \"predict_parameters\":{\n",
    "        \"drift_type\": \"feature\"\n",
    "      }\n",
    "    }\n",
    "  }\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Start serving our model\n",
    "\n",
    "Now that we have our config in-place, we can start the server by running mlserver start command. This needs to either be ran from the same directory where our config files are or pointing to the folder where they are.\n",
    "\n",
    "```shell\n",
    "mlserver start .\n",
    "```\n",
    "\n",
    "Since this command will start the server and block the terminal, waiting for requests, this will need to be ran in the background on a separate terminal."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Send test inference request\n",
    "\n",
    "We now have our alibi-detect model being served by `mlserver`. To make sure that everything is working as expected, let's send a request from our test set.\n",
    "\n",
    "For that, we can use the Python types that `mlserver` provides out of box, or we can build our request manually."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import requests\n",
    "\n",
    "inference_request = {\n",
    "    \"inputs\": [\n",
    "        {\n",
    "            \"name\": \"predict\",\n",
    "            \"shape\": X_t0.shape,\n",
    "            \"datatype\": \"FP32\",\n",
    "            \"data\": X_t0.tolist(),\n",
    "        }\n",
    "    ]\n",
    "}\n",
    "\n",
    "endpoint = \"http://localhost:8080/v2/models/income-tabular-drift/versions/v0.1.0/infer\"\n",
    "response = requests.post(endpoint, json=inference_request)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### View model response"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import json\n",
    "response_dict = json.loads(response.text)\n",
    "\n",
    "labels = ['No!', 'Yes!']\n",
    "for f in range(cd_tabular.n_features):\n",
    "    stat = 'Chi2' if f in list(categories_per_feature.keys()) else 'K-S'\n",
    "    fname = feature_names[f]\n",
    "    is_drift = response_dict['outputs'][0]['data'][f]\n",
    "    stat_val, p_val = response_dict['outputs'][1]['data'][f], response_dict['outputs'][2]['data'][f]\n",
    "    print(f'{fname} -- Drift? {labels[is_drift]} -- Chi2 {stat_val:.3f} -- p-value {p_val:.3f}')"
   ]
  }
 ],
 "metadata": {
  "interpreter": {
   "hash": "5377649fd9904e56b1a72eb79a8e3137babc0e642f8d1adb411c1391b7f9d00d"
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
